Active Imitation Learning via State Queries

We consider the problem of active imitation learning. In passive imitation learning, the goal is to learn a target policy by observing full trajectories of it. Unfortunately, generating such trajectories requires substantial effort and can be impractical in some cases. Active imitation learning reduces this effort by querying the teacher about individual states. Given such a query, the teacher may either suggest an action for the state or declare that the state is “bad” in the sense that the teacher’s policy would never go there. Standard active-learning techniques, do not account for the state-visitation likelihood of the target policy, and hence can perform poorly by asking many “bad” queries. We describe a new approach to this problem that is inspired by viewing query selection in the framework of Bayesian active learning, resulting in a version of Query-by-Committee for active imitation learning. Our experiments in two test domains show promise for our approach compared to a number of alternatives.